The present invention relates to image processing apparatus and image processing methods for processing a right viewpoint image and a left viewpoint image, or multi-viewpoint images, with parallax, as well as image processing systems.
It is known that a human being perceives the third dimension of an object by viewing the object with his/her right and left eyes at different angles and distances. The difference in appearance of the object as viewed with the right and left eyes is referred to as parallax, and the magnitude of the parallax is referred to as the amount of parallax (amount of vision disparity).
Up until today proposed were methods of exploiting the principle of third dimension perception by human beings to make people recognize two 2D images with parallax as a stereo image (3D image). As an example: If one and the same subject is shot at different angles from two viewpoints positioned right and left, respectively, to obtain a right viewpoint image and a left viewpoint image with parallax, and the right viewpoint image is presented to the right eye of a person, while the left viewpoint image is presented to the left eye, the person recognizes the two 2D images as one 3D image.
Since the third dimension perception by human beings is according to the magnitude of the parallax, a stereoscopic impression (or, the depth) of a 3D image can be modified by displacing right and left viewpoint images physically in horizontal directions to adjust the amount of parallax.
The amount of parallax is known to depend on the size of the 3D image to be displayed, with a larger size increasing the amount of parallax between a right viewpoint image and a left viewpoint image, and the 3D image depth as well. It is also known that a 3D image is presented as a superposition of a right viewpoint image and a left viewpoint image, and the superposition, that is to say, a portion displayable as a 3D image varies in range with the amount of parallax.
In the case of human beings, the distance between the right and left eyes (interpupillary distance) is about 65 mm on average, and such a distance is considered optimal as the distance between lenses for the conventional 3D shooting. In contrast, the assignee of the present application, FUJIFILM Corporation, has disclosed a stereoscopic camera (3D digital camera) in which the distance between two lenses used for taking right and left viewpoint images, respectively, is increased from the conventional one by ca. 20%. The camera as such somewhat enhances the 3D image depth in comparison with the third dimension of the subject that is perceived by human eyes. In addition, this stereoscopic camera has a parallax adjusting function capable of being toggled between automatic and manual modes, which makes it possible to take and appreciate a natural 3D image irrespective of the distance to the subject.
It is said that human beings receive a stereoscopic impression owing to “binocular parallax” caused by the differences between the right and left eyes in viewing angle and distance. For the LCD monitor of the stereoscopic camera as above, the 3D image displaying technologies which the assignee of the present application has developed on its own, such as “Light Direction Control System” and “Parallax Barrier System,” are employed. By the technologies, lights entering into the right and left eyes, respectively, are controlled in direction with high precision to project different images onto the two eyes, and thereby allow a natural, stereoscopic impression to be received with the naked eye.
The patent documents as mentioned below are prior art documents related to the present invention.
JP 4442190 B describes that a marginal region is provided on each of the right and left sides of a right viewpoint image, and an image for stereopsis is appropriately produced, with no areas thereof being made undisplayable, even if the right viewpoint image or a left viewpoint image is slid.
JP 8-205203 A describes that a 3D image is obtained by horizontally shifting a left eye image and a right eye image to the right and the left, respectively, by a specified number of pixels to thereby cause the entire image displayed to appear protruding forward from the display screen. It is also described in the patent document that the displayed image will lack a right end portion of a left eye image and a left end portion of a right eye image if left and right eye video signals are horizontally shifted.
JP 2003-331317 A describes that a parallax map expressing the depth distribution for a subject is prepared with respect to image comprised of a plurality of layers from information on areas selected in the individual layers, and on spacings between the individual layers, and multi-viewpoint images are generated from the parallax map and a 2D image.
JP 9-191393 A describes that, during the production of an image for stereopsis from two-dimensional images, the entire original may not be recorded on a sheet of recording paper due to the relationship among the sizes of the original and the sheet of paper, the scaling factor, and the amount of displacement, and, in that case, the event is made known on a display.
During photo printing of a 3D image taken with a stereoscopic camera, a printing order is received through an order receiving apparatus (shopfront order receptor) placed in an agency dealing with photo printing of 3D images, a webpage on the Internet or the like, and image data (3D image data) on the 3D image, order information, and so forth are sent to a laboratory for photo printing of 3D images. In the laboratory, the amount of parallax between a right viewpoint image and a left viewpoint image is manually adjusted by an operator to make the photo print in which the depth of the 3D image is modified.
When a 3D image is synthesized from a right viewpoint image and a left viewpoint image, the two viewpoint images are physically shifted in horizontal directions to adjust the amount of parallax and thereby modify the depth of the 3D image, as mentioned before. Since a 3D image is limited to a superposition of a right viewpoint image and a left viewpoint image, there may arise a problem with photo printing of a 3D image, that is to say, a photo print actually received by an orderer (user) may differ from that expected by the orderer in printed area depending on the degree of parallax adjustment.